Skip-gram Word2Vec

In this notebook, I'll lead you through using PyTorch to implement the Word2Vec algorithm using the skip-gram architecture. By implementing this, you'll learn about embedding words for use in natural language processing. This will come in handy when dealing with things like machine translation.

Readings

Here are the resources I used to build this notebook. I suggest reading these either beforehand or while you're working on this material.

A really good conceptual overview of Word2Vec from Chris McCormick
First Word2Vec paper from Mikolov et al.
Neural Information Processing Systems, paper with improvements for Word2Vec also from Mikolov et al.

Word embeddings

When you're dealing with words in text, you end up with tens of thousands of word classes to analyze; one for each word in a vocabulary. Trying to one-hot encode these words is massively inefficient because most values in a one-hot vector will be set to zero. So, the matrix multiplication that happens in between a one-hot input vector and a first, hidden layer will result in mostly zero-valued hidden outputs.

To solve this problem and greatly increase the efficiency of our networks, we use what are called embeddings. Embeddings are just a fully connected layer like you've seen before. We call this layer the embedding layer and the weights are embedding weights. We skip the multiplication into the embedding layer by instead directly grabbing the hidden layer values from the weight matrix. We can do this because the multiplication of a one-hot encoded vector with a matrix returns the row of the matrix corresponding the index of the "on" input unit.

Instead of doing the matrix multiplication, we use the weight matrix as a lookup table. We encode the words as integers, for example "heart" is encoded as 958, "mind" as 18094. Then to get hidden layer values for "heart", you just take the 958th row of the embedding matrix. This process is called an embedding lookup and the number of hidden units is the embedding dimension.

There is nothing magical going on here. The embedding lookup table is just a weight matrix. The embedding layer is just a hidden layer. The lookup is just a shortcut for the matrix multiplication. The lookup table is trained just like any weight matrix.

Embeddings aren't only used for words of course. You can use them for any model where you have a massive number of classes. A particular type of model called Word2Vec uses the embedding layer to find vector representations of words that contain semantic meaning.

Word2Vec

The Word2Vec algorithm finds much more efficient representations by finding vectors that represent the words. These vectors also contain semantic information about the words.

Words that show up in similar contexts, such as "coffee", "tea", and "water" will have vectors near each other. Different words will be further away from one another, and relationships can be represented by distance in vector space.

There are two architectures for implementing Word2Vec:

CBOW (Continuous Bag-Of-Words) and

Skip-gram

In this implementation, we'll be using the skip-gram architecture because it performs better than CBOW. Here, we pass in a word and try to predict the words surrounding it in the text. In this way, we can train the network to learn representations for words that show up in similar contexts.

Loading Data

Next, we'll ask you to load in data and place it in the data directory

Load the text8 dataset; a file of cleaned up Wikipedia article text from Matt Mahoney.
Place that data in the data folder in the home directory.
Then you can extract it and delete the archive, zip file to save storage space.

After following these steps, you should have one file in your data directory: data/text8.



In [1]:

    
# read in the extracted text file      
with open('data/text8') as f:
    text = f.read()

# print out the first 100 characters
print(text[:100])









    



 anarchism originated as a term of abuse first used against early working class radicals including t

Pre-processing

Here I'm fixing up the text to make training easier. This comes from the utils.py file. The preprocess function does a few things:

It converts any punctuation into tokens, so a period is changed to <PERIOD>. In this data set, there aren't any periods, but it will help in other NLP problems.

It removes all words that show up five or fewer times in the dataset. This will greatly reduce issues due to noise in the data and improve the quality of the vector representations.

It returns a list of words in the text.

This may take a few seconds to run, since our text file is quite large. If you want to write your own functions for this stuff, go for it!



In [2]:

    
import utils

# get list of words
words = utils.preprocess(text)
print(words[:30])









    



['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse', 'first', 'used', 'against', 'early', 'working', 'class', 'radicals', 'including', 'the', 'diggers', 'of', 'the', 'english', 'revolution', 'and', 'the', 'sans', 'culottes', 'of', 'the', 'french', 'revolution', 'whilst']



In [3]:

    
# print some stats about this word data
print("Total words in text: {}".format(len(words)))
print("Unique words: {}".format(len(set(words)))) # `set` removes any duplicate words









    



Total words in text: 16680599
Unique words: 63641

Dictionaries

Next, I'm creating two dictionaries to convert words to integers and back again (integers to words). This is again done with a function in the utils.py file. create_lookup_tables takes in a list of words in a text and returns two dictionaries.

The integers are assigned in descending frequency order, so the most frequent word ("the") is given the integer 0 and the next most frequent is 1, and so on.

Once we have our dictionaries, the words are converted to integers and stored in the list int_words.



In [4]:

    
vocab_to_int, int_to_vocab = utils.create_lookup_tables(words)
int_words = [vocab_to_int[word] for word in words]

print(int_words[:30])









    



[5233, 3080, 11, 5, 194, 1, 3133, 45, 58, 155, 127, 741, 476, 10571, 133, 0, 27349, 1, 0, 102, 854, 2, 0, 15067, 58112, 1, 0, 150, 854, 3580]

Subsampling

Words that show up often such as "the", "of", and "for" don't provide much context to the nearby words. If we discard some of them, we can remove some of the noise from our data and in return get faster training and better representations. This process is called subsampling by Mikolov. For each word $w_i$ in the training set, we'll discard it with probability given by

$$ P(w_i) = 1 - \sqrt{\frac{t}{f(w_i)}} $$

where $t$ is a threshold parameter and $f(w_i)$ is the frequency of word $w_i$ in the total dataset.

$$ P(0) = 1 - \sqrt{\frac{1*10^{-5}}{1*10^6/16*10^6}} = 0.98735 $$

I'm going to leave this up to you as an exercise. Check out my solution to see how I did it.

Exercise: Implement subsampling for the words in int_words. That is, go through int_words and discard each word given the probablility $P(w_i)$ shown above. Note that $P(w_i)$ is the probability that a word is discarded. Assign the subsampled data to train_words.



In [5]:

    
from collections import Counter
import random
import numpy as np

threshold = 1e-5
word_counts = Counter(int_words)
#print(list(word_counts.items())[0])  # dictionary of int_words, how many times they appear

total_count = len(int_words)
freqs = {word: count/total_count for word, count in word_counts.items()}
p_drop = {word: 1 - np.sqrt(threshold/freqs[word]) for word in word_counts}
# discard some frequent words, according to the subsampling equation
# create a new list of words for training
train_words = [word for word in int_words if random.random() < (1 - p_drop[word])]

print(train_words[:30])









    



[5233, 3080, 58, 741, 10571, 27349, 15067, 58112, 3580, 190, 10712, 6, 104, 2731, 3672, 708, 53, 7088, 5233, 1052, 44611, 2877, 5233, 8983, 4147, 6437, 4186, 362, 344, 1818]

Making batches

Now that our data is in good shape, we need to get it into the proper form to pass it into our network. With the skip-gram architecture, for each word in the text, we want to define a surrounding context and grab all the words in a window around that word, with size $C$.

From Mikolov et al.:

"Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples... If we choose $C = 5$, for each training word we will select randomly a number $R$ in range $[ 1: C ]$, and then use $R$ words from history and $R$ words from the future of the current word as correct labels."

Exercise: Implement a function get_target that receives a list of words, an index, and a window size, then returns a list of words in the window around the index. Make sure to use the algorithm described above, where you chose a random number of words to from the window.

Say, we have an input and we're interested in the idx=2 token, 741:

[5233, 58, 741, 10571, 27349, 0, 15067, 58112, 3580, 58, 10712]

For R=2, get_target should return a list of four values:

[5233, 58, 10571, 27349]



In [6]:

    
def get_target(words, idx, window_size=5):
    ''' Get a list of words in a window around an index. '''
    
    R = np.random.randint(1, window_size+1)
    start = idx - R if (idx - R) > 0 else 0
    stop = idx + R
    target_words = words[start:idx] + words[idx+1:stop+1]
    
    return list(target_words)



In [7]:

    
# test your code!

# run this cell multiple times to check for random window selection
int_text = [i for i in range(10)]
print('Input: ', int_text)
idx=5 # word index of interest

target = get_target(int_text, idx=idx, window_size=5)
print('Target: ', target)  # you should get some indices around the idx









    



Input:  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Target:  [3, 4, 6, 7]

Generating Batches

Here's a generator function that returns batches of input and target data for our model, using the get_target function from above. The idea is that it grabs batch_size words from a words list. Then for each of those batches, it gets the target words in a window.



In [8]:

    
def get_batches(words, batch_size, window_size=5):
    ''' Create a generator of word batches as a tuple (inputs, targets) '''
    
    n_batches = len(words)//batch_size
    
    # only full batches
    words = words[:n_batches*batch_size]
    
    for idx in range(0, len(words), batch_size):
        x, y = [], []
        batch = words[idx:idx+batch_size]
        for ii in range(len(batch)):
            batch_x = batch[ii]
            batch_y = get_target(batch, ii, window_size)
            y.extend(batch_y)
            x.extend([batch_x]*len(batch_y))
        yield x, y



In [9]:

    
int_text = [i for i in range(20)]
x,y = next(get_batches(int_text, batch_size=4, window_size=5))

print('x\n', x)
print('y\n', y)









    



x
 [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3]
y
 [1, 2, 3, 0, 2, 3, 0, 1, 3, 1, 2]

Building the graph

Below is an approximate diagram of the general structure of our network.

The input words are passed in as batches of input word tokens.

This will go into a hidden layer of linear units (our embedding layer).

Then, finally into a softmax output layer.

We'll use the softmax layer to make a prediction about the context words by sampling, as usual.

The idea here is to train the embedding layer weight matrix to find efficient representations for our words. We can discard the softmax layer because we don't really care about making predictions with this network. We just want the embedding matrix so we can use it in other networks we build using this dataset.

Validation

Here, I'm creating a function that will help us observe our model as it learns. We're going to choose a few common words and few uncommon words. Then, we'll print out the closest words to them using the cosine similarity:

$$ \mathrm{similarity} = \cos(\theta) = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}||\vec{b}|} $$

We can encode the validation words as vectors $\vec{a}$ using the embedding table, then calculate the similarity with each word vector $\vec{b}$ in the embedding table. With the similarities, we can print out the validation words and words in our embedding table semantically similar to those words. It's a nice way to check that our embedding table is grouping together words with similar semantic meanings.



In [11]:

    
def cosine_similarity(embedding, valid_size=16, valid_window=100, device='cpu'):
    """ Returns the cosine similarity of validation words with words in the embedding matrix.
        Here, embedding should be a PyTorch embedding module.
    """
    
    # Here we're calculating the cosine similarity between some random words and 
    # our embedding vectors. With the similarities, we can look at what words are
    # close to our random words.
    
    # sim = (a . b) / |a||b|
    
    embed_vectors = embedding.weight
    
    # magnitude of embedding vectors, |b|
    magnitudes = embed_vectors.pow(2).sum(dim=1).sqrt().unsqueeze(0)
    
    # pick N words from our ranges (0,window) and (1000,1000+window). lower id implies more frequent 
    valid_examples = np.array(random.sample(range(valid_window), valid_size//2))
    valid_examples = np.append(valid_examples,
                               random.sample(range(1000,1000+valid_window), valid_size//2))
    valid_examples = torch.LongTensor(valid_examples).to(device)
    
    valid_vectors = embedding(valid_examples)
    similarities = torch.mm(valid_vectors, embed_vectors.t())/magnitudes
        
    return valid_examples, similarities

SkipGram model

Define and train the SkipGram model.

You'll need to define an embedding layer and a final, softmax output layer.

An Embedding layer takes in a number of inputs, importantly:

num_embeddings – the size of the dictionary of embeddings, or how many rows you'll want in the embedding weight matrix
embedding_dim – the size of each embedding vector; the embedding dimension



In [1]:

    
import torch
from torch import nn
import torch.optim as optim



In [12]:

    
class SkipGram(nn.Module):
    def __init__(self, n_vocab, n_embed):
        super().__init__()
        
        self.embed = nn.Embedding(n_vocab, n_embed)
        self.output = nn.Linear(n_embed, n_vocab)
        self.log_softmax = nn.LogSoftmax(dim=1)
    
    def forward(self, x):
        x = self.embed(x)
        scores = self.output(x)
        log_ps = self.log_softmax(scores)
        
        return log_ps

Training

Below is our training loop, and I recommend that you train on GPU, if available.

Note that, because we applied a softmax function to our model output, we are using NLLLoss as opposed to cross entropy. This is because Softmax in combination with NLLLoss = CrossEntropy loss .



In [13]:

    
# check if GPU is available
device = 'cuda' if torch.cuda.is_available() else 'cpu'

embedding_dim=300 # you can change, if you want

model = SkipGram(len(vocab_to_int), embedding_dim).to(device)
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

print_every = 500
steps = 0
epochs = 5

# train for some number of epochs
for e in range(epochs):
    
    # get input and target batches
    for inputs, targets in get_batches(train_words, 512):
        steps += 1
        inputs, targets = torch.LongTensor(inputs), torch.LongTensor(targets)
        inputs, targets = inputs.to(device), targets.to(device)
        
        log_ps = model(inputs)
        loss = criterion(log_ps, targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if steps % print_every == 0:                  
            # getting examples and similarities      
            valid_examples, valid_similarities = cosine_similarity(model.embed, device=device)
            _, closest_idxs = valid_similarities.topk(6) # topk highest similarities
            
            valid_examples, closest_idxs = valid_examples.to('cpu'), closest_idxs.to('cpu')
            for ii, valid_idx in enumerate(valid_examples):
                closest_words = [int_to_vocab[idx.item()] for idx in closest_idxs[ii]][1:]
                print(int_to_vocab[valid_idx.item()] + " | " + ', '.join(closest_words))
            print("...")









    



its | apothecary, entourage, kapoor, captives, perched
or | bridgetown, surveyed, pistis, engages, tiebreaker
as | madhava, messina, trollh, realises, gibraltarians
be | ganymede, matti, dek, competition, gillespie
may | propose, mesh, trapani, lycaon, nod
more | thracians, informing, blvd, failures, aes
and | returns, liverpudlians, limbs, redistributed, operative
who | tackles, discretization, lexically, bleed, kippur
engine | khorasan, candlemas, flemings, strasberg, israelites
channel | poetry, debugging, needed, milla, attracts
operating | taoism, asch, extensions, starting, piccard
bbc | highwind, demigods, complications, atl, temples
alternative | hazmi, dh, lecturers, asthma, ragged
paris | bil, phenomenal, tonight, thugs, accumulations
lived | hartsfield, transcription, buddhism, ancestral, ldap
question | meats, colossians, communism, mwh, theophany
...
after | marital, hero, strongholds, lavey, protrude
four | locker, correction, mocked, screenwriter, itinerarium
into | markka, systematised, unionised, ningen, westminsters
seven | turkey, grandfather, southampton, diz, staining
years | defense, poland, rescinded, maio, silence
first | bedfordshire, levitated, telemann, congested, porn
not | agadir, entertainers, ashtabula, squares, oil
they | harmonica, mainstream, wedded, theodor, seagal
shown | organises, squid, quanta, eads, canetti
universe | xa, fauna, infective, personified, chiral
know | dropkick, jabir, taxis, console, collaborations
egypt | vu, semicircle, donates, connection, durga
construction | vivid, openstep, assaulted, penny, zeng
rise | sextant, aguilera, bored, deviate, abrasion
discovered | tractate, sweating, lithographs, ditions, metaphor
ice | decins, horned, lb, btr, invites
...
was | sterile, sleeker, torts, bn, knowledge
a | rond, bornemisza, boyne, sewerage, fielded
one | zero, aymara, alvis, vibrates, charlevoix
its | apothecary, entourage, captives, kapoor, hommes
while | broca, incapacity, stratification, kernels, went
not | entertainers, agadir, ashtabula, fundamentalism, squares
will | lfheim, needles, disuse, deficiencies, rally
time | scientifiques, liquidated, funds, judges, viewing
instance | formalizing, disliking, breviaries, moselle, voyage
defense | ignatius, years, pazzi, footnotes, goatskin
liberal | tibor, deposited, lesbians, phenols, msi
rise | sextant, aguilera, deviate, bored, shreck
event | ninotchka, grupo, anthropic, sooty, baudot
universe | xa, fauna, infective, personified, chiral
creation | ali, diomedea, catullus, olney, watashi
award | bombers, hold, arslan, briar, morant
...
the | genji, gentry, crypto, crushed, vehicle
been | housed, eggs, fossil, erne, colvin
some | admetus, outlays, oscilloscope, lachine, baryons
often | transpersonal, viracocha, coffers, defibrillator, structured
would | novial, ux, grumpy, laumann, chan
from | nederlandse, grammatical, combed, exploiting, mod
more | thracians, informing, wondrous, maturation, trace
that | committing, jaffa, pods, tardive, mirrored
except | fuga, hermitian, immigrating, cramped, diplomats
dr | gokturk, midwinter, glimpsed, untethered, ziggurat
articles | antinomies, kondratiev, normandie, gong, scored
troops | refusal, bird, hamm, toile, hornblende
instance | formalizing, disliking, breviaries, moselle, voyage
operating | piccard, starting, extensions, origami, taoism
accepted | unsatisfactory, beats, harboring, cervix, sigismund
defense | years, ignatius, footnotes, pazzi, sayers
...
history | throws, famer, bionic, ignacy, displeased
such | expeditions, lpc, kumasi, wes, ifrcs
after | hero, marital, strongholds, uncontrollable, latterly
not | fulda, entertainers, squares, ashtabula, agadir
its | apothecary, covered, captives, entourage, hommes
been | housed, colvin, involves, fossil, limitation
would | grumpy, novial, ux, chan, laumann
will | needles, lfheim, deficiencies, marquardt, jmu
older | meiosis, bent, szeged, geomorphology, thrills
test | popcorn, guessing, sapkowski, infiltrate, ointment
shows | boudin, begining, abkhazians, synchrotron, reputation
universe | xa, fauna, personified, chiral, infective
egypt | durga, semicircle, connection, vu, donates
active | dcc, cheng, presides, universes, follower
animals | burghers, ventura, foix, relish, neonatal
applied | sonar, perforated, server, flowchart, mcarthur
...
the | genji, gentry, crypto, crushed, wield
only | khazaria, tournament, customize, commended, qazaqstan
used | ische, animator, anthrax, harming, singularities
world | hammond, angled, giantesses, galen, dallas
b | colliery, l, diesel, physician, stalemate
two | iuds, hamza, volsunga, byng, per
other | accessible, mistral, convertible, evesham, engrailed
some | diocese, admetus, oscilloscope, outlays, lachine
know | dropkick, jabir, console, collaborations, sectarianism
cost | patass, pile, orphan, piracy, transmitting
nobel | maceo, gambler, wilkes, suspends, bus
existence | kovacs, conway, bats, saaremaa, reminiscent
bbc | highwind, demigods, atl, counselling, temples
professional | ousted, warmer, dismisses, havok, agitated
applications | polygamous, watercourses, os, chaining, voroshilov
operating | extensions, piccard, origami, sleepers, starting
...
not | fulda, squares, agadir, oil, ashtabula
people | unfettered, arachnoid, admittance, offenders, deaths
been | colvin, housed, horseback, liberties, refutations
american | berglin, waterman, cassie, agricola, vivant
so | xs, schedules, mah, mengistu, intrinsic
eight | five, bosman, commissioning, sedating, churn
will | deficiencies, needles, jmu, sinning, disoriented
on | nuances, freezer, externalized, tumbling, nyi
consists | arpa, kilobytes, zyklon, reluctance, florin
issue | tracked, trotskyist, franchises, epistemology, banknotes
heavy | prerequisite, dionne, lifeform, remonstrance, housed
universe | xa, personified, fauna, kill, infective
freedom | schuyler, moltke, political, elite, rationalize
grand | dazzle, archaeology, disappointment, motu, invitation
additional | faithfulness, tactical, sedating, gastronomy, conlangs
behind | milo, impotent, biographical, ucd, kfar
...
have | apps, damasio, owning, virginity, rerun
were | niklas, whit, aelia, sorghum, mitre
united | pharmaceutical, romanov, reuben, corporation, americas
these | dividend, paulaner, grading, fissure, exempt
a | rond, fair, paige, pendleton, ulster
had | statements, wirtschaftswunder, merry, iga, condescending
has | apulia, cody, nevers, archaelogical, jes
where | honours, bill, duplicity, hashihito, babel
ice | decins, salty, horned, burmese, btr
engineering | courageous, perplexing, criture, sensibilities, interrupts
additional | faithfulness, alkaline, sedating, tactical, conlangs
question | colossians, meats, communism, theophany, mwh
prince | pyroxene, engravings, josephine, aquileia, fraser
pre | youngman, preferential, travelin, achill, coles
applied | sonar, perforated, smoot, chimp, technologists
discovered | chemistry, sweating, metaphor, pavlova, lithographs
...
for | isabelle, malayo, rally, lg, staggered
be | sayings, defencemen, sparkle, clone, appreciated
into | systematised, unionised, conjugacy, markka, segment
was | bn, resentful, homogenic, azores, sepp
has | apulia, cody, ifab, archaelogical, nevers
will | jmu, disoriented, deficiencies, needles, tolerable
where | hashihito, reminding, bill, akkad, weald
known | eisenman, mosaic, emedicine, wls, convocation
cost | patass, pile, piracy, coleco, transmitting
shown | squid, majlis, canetti, organises, nonhuman
rise | aguilera, sextant, deviate, according, modulators
nobel | maceo, hanlon, gambler, suspends, wilkes
orthodox | corrupt, waived, bude, lecturers, silt
articles | normandie, kondratiev, antinomies, amigaos, ramstein
construction | vivid, cubewanos, openstep, penny, unsuited
ocean | archeologists, outcrops, nepomuk, jinn, steller
...
the | gentry, genji, unskilled, lionhead, crypto
while | incapacity, broca, marxists, feyd, stratification
as | allocates, franz, statements, authored, madhava
would | novial, their, grumpy, theologico, centres
from | nederlandse, coates, combed, polonia, grammatical
called | trilobite, aneurin, probed, proximity, struggling
he | cousin, abelard, spacey, nsaid, savage
use | conquistadors, cruel, stokes, alike, ves
bible | balboa, islam, fullness, tiananmen, wissowa
lived | hartsfield, ancestral, alamein, regarded, chroma
quite | kanpur, pastors, ooc, acclaim, hai
assembly | fabricius, rquez, crc, mughal, debayle
professional | ousted, dogme, unpublished, revamping, dismisses
nobel | maceo, hanlon, gambler, suspends, prize
liberal | tibor, commercial, lesbians, colonials, msi
additional | faithfulness, tactical, conlangs, alkaline, sedating
...
only | tournament, ribao, commended, customize, progeny
can | geodesic, suffix, terse, pixel, optimal
new | sections, soso, coburn, rabia, replica
if | dictator, butyric, wentworth, penal, improves
th | marwan, yeshiva, burgundian, newspaperman, eloheinu
world | siddha, hammond, played, giantesses, effort
than | nuisance, sigmoid, unreliability, makes, surged
five | eight, unusually, four, dynasty, kandinsky
freedom | political, rationalize, moltke, restoring, ancestries
primarily | pattie, uneven, have, testimonials, vytautas
animals | burghers, ventura, victims, relish, neonatal
bible | balboa, islam, wissowa, fullness, concr
mean | reformation, revolve, commences, catechumen, girl
troops | refusal, toile, jaspers, paolo, odense
behind | ucd, drinkers, impotent, drowns, milo
mathematics | trilogy, soci, dupuis, amputation, depressing
...
seven | two, zero, southampton, canvases, ing
when | madre, ucs, snatched, shorebirds, cordial
all | therapeutae, convair, therus, renovate, larsson
united | pharmaceutical, romanov, reuben, corporation, manhunter
so | schedules, mah, intrinsic, xs, mengistu
one | zero, nine, six, two, eight
they | clashed, crosley, inward, discretion, remuneration
years | poland, xry, rescinded, claret, defense
know | dropkick, jabir, console, collaborations, pick
file | rijndael, mellows, erlang, eben, dih
ice | decins, horned, salty, iris, clements
resources | balaton, pataki, clearer, exploration, sociopolitical
cost | patass, piracy, coleco, pile, sustaining
articles | normandie, kondratiev, antinomies, causeway, agu
defense | partisan, sayers, ignatius, cottle, pazzi
consists | arpa, aleksey, kilobytes, reluctance, nagaoka
...
be | defencemen, scythians, sayings, absence, homilies
only | tournament, ribao, commended, progeny, customize
were | centuries, sorghum, nahar, fc, niklas
often | transpersonal, viracocha, coerce, besides, interchangeably
people | deaths, arachnoid, nonproliferation, admittance, remember
to | bring, bentine, visigoth, jewish, breakthrough
have | primarily, rerun, owning, virginity, apps
a | rond, fair, presided, wizards, delayed
accepted | auditioning, unsatisfactory, sinned, arbenz, incantation
smith | satirists, deut, burman, chee, departing
consists | arpa, kilobytes, reluctance, aleksey, nagaoka
question | colossians, meats, theophany, mwh, proselytizing
taking | bimmer, ill, gyroscope, maslow, schwitters
centre | condenses, xyz, salvatore, tyr, touring
road | airport, vince, bubbled, imprisoning, protons
active | presides, dcc, cheng, transparency, savard
...
six | one, three, four, eight, two
over | hotels, gasp, baltic, congregated, sabbatical
are | there, lds, swnts, domed, vindicated
during | fripp, thunderstorm, gta, latinized, actinium
been | colvin, horseback, liberties, coria, colliding
of | london, in, now, spy, alsatian
however | panarion, kitchener, moderately, multipoint, wycliffe
world | siddha, giantesses, hammond, effort, arpa
pre | youngman, travelin, yoho, programmable, multitrack
san | chairman, pauling, modernized, muddy, foote
primarily | have, uneven, emc, pattie, testimonials
freedom | political, moltke, restoring, rationalize, tenured
writers | itch, abrahams, politicians, dave, centime
know | dropkick, console, jabir, collaborations, i
marriage | borja, aunt, withdrawal, stood, dissuade
bible | balboa, px, storefront, lddc, flyweight
...
had | condescending, statements, amnesiac, buddhism, iga
years | poland, claret, xry, rescinded, birth
if | butyric, chinon, expresses, nobody, oppressed
one | zero, nine, six, eight, two
use | conquistadors, cruel, uses, dictionary, scarab
world | hammond, siddha, effort, denslow, giantesses
called | proximity, cena, escorted, aneurin, elegans
when | madre, monadic, ucs, judicially, snatched
cost | patass, piracy, pile, coleco, insurrectionary
assembly | rquez, resigned, chief, crc, memo
writers | politicians, abrahams, itch, dave, by
magazine | overbearing, cigar, unfolding, singularity, hobby
animals | burghers, reproduction, victims, relish, neonatal
professional | ousted, players, unpublished, artie, revamping
powers | nationalists, bobcat, filo, generalship, conveys
channel | backward, tier, needed, attracts, smelling
...
may | starley, hostages, tastes, propose, merging
united | reuben, romanov, pharmaceutical, americas, corporation
about | waving, tumbuka, dmt, recessions, patmos
only | tournament, commended, ribao, modulated, customize
war | ii, saw, estas, defecting, mossad
for | isabelle, covering, most, outranks, lg
he | his, spacey, him, revue, doubts
first | dnia, bedfordshire, telemann, insensible, philistines
san | chairman, pauling, modernized, foote, conservatory
applied | perforated, technologists, sonar, smoot, chimp
channel | backward, radio, tier, milla, attracts
engineering | courageous, perplexing, soc, kak, arachne
applications | os, chaining, polygamous, watercourses, sht
bbc | highwind, updated, atl, computing, supplemented
orthodox | corrupt, adherents, bude, calvinists, chaldees
bill | sides, security, lakshadweep, lvaro, mason
...
five | four, eight, two, zero, six
who | kazakh, amram, alabama, shred, tackles
eight | one, five, four, six, two
war | ii, saw, defecting, troop, estas
as | participated, precautionary, trollh, iit, example
not | exclusively, fulda, could, invalidating, arcane
to | persson, bring, know, uproar, commercialize
many | online, viridis, gol, distinct, unquestioned
recorded | cello, xxxix, shark, wart, mcghee
operations | eigenfunctions, egress, hemorrhagic, ownership, landholding
bill | sides, lakshadweep, mason, lvaro, security
proposed | tariff, janus, reinstate, governorates, formulation
hold | comanche, muad, prem, homeported, visualized
discovered | chemistry, lithographs, pavlova, peach, sweating
know | dropkick, tried, jabir, console, collaborations
notes | redirection, wiki, vma, octave, ravens
...
known | minor, sahel, settled, frustrating, emedicine
up | iommi, corbett, encryption, dario, erlewine
use | uses, iec, conquistadors, hyperplasia, dimethyl
about | tumbuka, waving, geoid, recessions, cumbric
seven | two, nine, zero, one, eight
called | covalent, cena, mu, impedance, escorted
between | conceptual, barrichello, abreast, rude, circumference
not | could, or, accept, exclusively, invalidating
question | colossians, proselytizing, meats, theophany, emphasizing
lived | hartsfield, ancestral, alamein, chroma, transcription
running | layering, runs, federer, halacha, tokelau
magazine | overbearing, cigar, kino, collectible, hobby
bbc | highwind, computing, updated, atl, zero
articles | normandie, reformation, gong, kondratiev, spiky
taking | ill, joe, maslow, gyroscope, annalen
file | rijndael, psychoanalytical, mellows, grothendieck, mesi
...
for | kickoff, isabelle, chambersburg, blacked, pacos
people | denial, deaths, remember, stress, admittance
american | recipient, singer, actress, swedish, nine
most | are, chih, deformation, polytheistic, modula
no | clique, fraktion, ingenioso, user, reelection
his | he, was, her, him, relics
the | of, hades, range, first, to
other | cations, lymphedema, is, rinse, ic
grand | dazzle, street, archaeology, motu, nc
proposed | tariff, reinstate, governorates, janus, reluctantly
older | plantes, rampton, lexical, ousterhout, extravagance
quite | kanpur, ooc, acclaim, pastors, recipe
behind | drowns, vecchio, archelaus, spyglass, girondins
troops | refusal, army, battle, wwi, armies
san | conservatory, chairman, pauling, modernized, upriver
know | dropkick, tried, jabir, i, sectarianism
...
this | datatype, supportive, submission, paraclete, generally
where | bypasses, zenith, tajiks, perpendicular, weald
or | word, and, milnor, seaweed, medial
people | deaths, denial, remember, admittance, presidencia
five | four, eight, one, zero, six
also | is, usp, ability, ethnicity, pine
for | isabelle, hotspur, kickoff, antiques, in
of | religious, in, azad, the, now
http | page, html, loosely, www, web
san | conservatory, chairman, pauling, modernized, plaid
older | plantes, rampton, ousterhout, extravagance, lexical
rise | sextant, deviate, aguilera, modulators, according
file | rijndael, psychoanalytical, executables, mellows, ignoble
except | planners, asgard, veiled, japonic, choked
hold | taxpayers, unthinkable, comanche, prem, demarcated
derived | melos, kilowatt, ferment, pd, progressives
...
some | recognize, admetus, dentists, gaon, marxian
other | spearman, or, lymphedema, is, number
used | stroking, ische, harming, typographical, particularly
where | bypasses, hades, tajiks, weald, zenith
from | satsuki, iban, magic, griffon, jell
or | word, milnor, medial, other, surveyed
the | and, of, in, wares, servicing
world | played, denslow, effort, belgium, in
construction | cubewanos, vivid, behaviours, building, massively
scale | greyish, nueva, miscarriages, bewildering, horst
assembly | memo, chief, appointed, interchanged, re
mean | catechumen, unobservable, girl, revolve, reformation
http | page, html, www, loosely, web
pope | xi, john, kitchener, hypertalk, expedients
behind | drowns, archelaus, spyglass, stumbling, would
something | statements, refreshed, ytterbium, crazy, numb
...
to | not, tariff, ministries, the, in
there | are, evangelical, ih, males, mong
be | that, defencemen, they, this, any
system | dortmund, peripheral, developments, altes, gc
into | systematised, dissented, turbines, paternoster, absorbing
people | deaths, denial, remember, exterminated, admittance
all | are, existence, if, invertible, or
world | denslow, effort, bud, bader, bolivia
active | presides, salut, transparency, recommendations, bodies
notes | icao, wiki, vma, tswana, redirection
bbc | highwind, radio, interview, updated, naughty
magazine | intergalactic, overbearing, cigar, kino, gwen
joseph | morris, jeffery, sam, pastoral, moreau
numerous | malibu, enunciation, isolation, clientele, roaming
derived | kilowatt, pd, melos, ferment, polycystic
construction | cubewanos, behaviours, vivid, mitcham, dickens
...
d | b, kurt, mprp, yx, michel
will | my, your, they, you, looks
time | externalization, mael, funds, extrapolating, thermal
had | his, was, he, him, condescending
most | have, for, are, deformation, chih
often | layers, interchangeably, coerce, sketching, fainting
is | the, a, or, also, to
was | his, had, were, salary, nikolai
pressure | unconsciousness, transition, conduction, kinetic, quarters
orthodox | church, catholic, denomination, interests, corrupt
lived | hartsfield, ancestral, alamein, chroma, frost
centre | touring, xyz, varuna, condenses, salvatore
grand | street, motu, dazzle, highwind, invitation
test | popcorn, icao, guessing, infiltrate, niosh
marriage | aunt, upheld, kazantzakis, borja, migrated
shown | exponential, realization, ascertain, extensionality, unprotected
...
people | deaths, exterminated, remember, ones, natives
three | one, zero, two, four, six
where | bypasses, sides, recoiled, if, will
have | some, most, are, capacities, their
states | individuals, international, amendment, entailed, civil
by | the, wonderland, accountants, was, ashton
for | most, a, duplicated, how, persuade
known | games, also, minor, mythology, loosely
frac | we, iy, y, theta, constant
universe | xa, personified, origin, supernatural, bo
bill | security, mason, lakshadweep, sides, lvaro
account | fornax, real, farmhouse, foreshadow, lasor
versions | ess, lx, systems, starcraft, erina
grand | motu, street, dazzle, invitation, anticlockwise
channel | network, radio, backward, debugging, hare
construction | cubewanos, vivid, behaviours, mitcham, building
...
are | there, most, of, and, have
an | isosceles, mcfadden, dom, additional, coucy
were | anticipated, reassert, have, sorghum, been
many | distinct, sounds, unquestioned, related, are
while | entertain, corkscrew, incapacity, phospholipid, dune
of | the, and, in, a, are
four | one, two, five, three, six
over | total, two, baltic, one, male
pre | preferential, eliyahu, travelin, vedas, youngman
woman | daughter, male, marry, live, pregnant
stage | actors, singled, walk, patton, concert
ocean | himalaya, outcrops, intelsat, sierra, pillar
joseph | jeffery, morris, sam, mortimer, honor
alternative | hazmi, dh, criticized, bambara, earthenware
resources | diachronic, improved, spammers, agriculture, exploration
arts | art, graduate, susskind, school, institute
...
and | the, of, caracal, are, some
an | of, a, topological, is, algebra
th | nd, century, centuries, one, ceawlin
are | have, most, other, more, and
in | of, was, scandinavia, the, and
than | sigmoid, makes, nuisance, bigger, much
by | sussman, of, and, the, be
there | are, ih, cases, males, locality
stage | actors, walk, singled, beethoven, hollywood
san | conservatory, caldera, cavallo, modernized, earn
active | combustible, salut, presides, transparency, excludes
consists | nagaoka, aleksey, bayh, reluctance, kilobytes
operations | operation, security, excision, landholding, eac
mean | unobservable, mantras, is, revolve, difference
bill | security, lvaro, mason, fun, comedian
governor | supreme, lord, forza, parliaments, democrat
...
at | noone, mile, angeles, finishing, per
zero | two, three, four, six, eight
or | is, word, other, to, not
will | my, looks, you, enough, switchable
some | have, are, as, such, other
may | vocative, starley, appropriate, insensible, or
s | nine, one, and, four, two
see | list, jagan, links, here, also
alternative | hazmi, dh, criticized, immemorial, bambara
troops | war, army, refusal, rebels, battle
writers | novelists, politicians, humorists, alumni, homepage
issue | banknotes, trotskyist, franchises, epistemology, goering
applied | technologists, intention, perforated, versioning, eec
mainly | balkan, patagonian, primarily, retained, beit
proposed | tariff, be, governorates, reinstate, hyperplane
smith | jim, paterson, chee, kaufman, lap
...
for | how, four, in, are, equating
an | intermembrane, of, inhumane, to, renewed
other | are, or, usually, some, is
in | the, of, world, after, and
are | there, other, such, most, or
up | corbett, iommi, saimiri, chatti, edge
system | systems, dortmund, separate, developments, development
war | military, troops, wwii, wars, defecting
hit | puente, microphone, restart, appropriate, wizard
engine | powered, fv, cars, glock, flemings
except | planners, is, parts, asgard, other
liberal | conservatism, msi, reform, commercial, argue
institute | arts, school, genealogy, studies, abrahams
egypt | egyptian, nasser, syria, depopulation, abdullah
units | amistad, kilogram, min, prefixes, disengage
dr | sara, giulia, nonaligned, trevor, aldrin
...
there | are, that, sabotaged, similar, insular
that | be, it, not, to, of
were | was, in, some, after, been
between | the, in, linea, seeks, both
where | recoiled, left, stitch, course, exists
for | in, to, tutelage, how, as
world | denslow, international, in, war, mtu
so | be, to, ourselves, intrinsic, nullified
hold | backstory, muad, unthinkable, cavalli, choosing
smith | jim, paterson, chee, satirists, kaufman
road | airport, roads, passenger, route, interstate
dr | giulia, sara, nonaligned, der, theodor
placed | trecento, canvas, excavation, defecting, laying
notes | compositions, redirection, tswana, octave, metrology
account | real, fornax, farmhouse, mench, impression
behind | archelaus, quickly, spyglass, skirmishing, drowns
...
this | a, the, is, or, it
when | it, in, sow, tta, that
had | was, until, brought, to, his
however | that, so, not, is, allies
will | my, you, allows, be, they
its | the, tourism, of, a, has
to | the, that, of, and, in
of | the, and, by, a, is
mainly | primarily, also, balkan, language, patagonian
hold | muad, taxpayers, unthinkable, backstory, inscrutable
report | rajendra, approved, june, biggio, office
applied | technologists, perforated, versioning, mesmer, chimp
existence | enlightenment, lihou, eternal, faiths, hominin
cost | price, piracy, sustaining, insurrectionary, ammunition
test | detection, rocket, popcorn, niosh, marginalize
stage | actors, radiate, beethoven, eckenstein, walk
...
there | are, not, that, in, go
other | are, is, or, these, some
four | one, two, five, three, seven
such | many, are, some, other, that
known | settled, mythology, mentions, of, ranks
see | list, links, external, also, coterminous
nine | one, seven, two, four, three
been | were, was, hashemite, committee, early
http | www, html, com, links, web
square | headwaters, hour, kilometre, area, miles
cost | price, piracy, invest, insurrectionary, sustaining
question | proselytizing, colossians, i, gentry, inasmuch
centre | sprawling, touring, town, innovations, city
rise | sextant, according, modulators, mieszko, caliente
behind | archelaus, quickly, spyglass, collapses, hold
pressure | kinetic, wick, unconsciousness, duma, lecithin
...
he | his, him, was, had, moved
which | the, and, by, tz, on
often | sometimes, rather, gamelan, horns, namo
use | used, manual, uses, available, admittedly
when | it, tta, that, then, landowners
the | in, is, and, of, its
a | is, and, the, of, into
up | to, you, i, corbett, the
centre | education, city, sprawling, town, campus
square | headwaters, hour, area, miles, kilometre
older | median, families, plantes, age, income
hold | taxpayers, backstory, millennialism, muad, unthinkable
test | rocket, detection, icao, popcorn, flight
file | storage, rijndael, psychoanalytical, mellows, mesi
running | runs, platform, vt, tokelau, coles
experience | hinduism, panpsychism, bodhi, teach, me
...
but | to, that, seamen, only, true
first | the, in, place, time, second
more | are, less, tend, such, for
many | such, are, some, have, most
years | zero, four, age, five, six
however | allies, that, not, necessarily, required
it | that, is, doing, do, not
united | states, nations, presidents, virginia, australia
governor | state, government, representative, democrat, cabinet
instance | be, lattices, cobol, formalizing, disliking
road | airport, roads, route, passenger, railway
hit | puente, hits, restart, album, cried
accepted | auditioning, colleague, athanasian, learn, fathers
ocean | atlantic, am, stations, pacific, intelsat
derived | root, glycosides, variants, word, kilowatt
taking | joe, ill, indistinguishable, involving, maslow
...
six | four, three, two, five, one
one | nine, eight, seven, four, three
five | four, two, eight, one, six
not | to, do, it, that, there
may | jurisdictions, any, conclusively, severity, lasts
two | three, four, zero, five, six
th | century, nd, late, six, centuries
american | eight, nine, born, singer, actor
mainly | primarily, included, sultanates, have, or
mathematics | mathematical, humanities, physics, study, mathematicians
placed | defecting, canvas, extensive, nairobi, trecento
dr | sara, cricketer, giulia, muggs, nonaligned
report | approved, rajendra, terrorists, biggio, outgrabe
professional | players, mascot, cornerbacks, revamping, sports
marriage | marriages, marry, divorce, married, aunt
resources | resource, petroleum, witwatersrand, country, agriculture
...
these | are, can, most, namespaces, several
also | are, see, and, as, is
can | be, are, ability, common, do
see | links, list, also, external, uses
from | the, on, by, and, to
known | of, also, the, is, nil
his | he, him, her, returned, had
to | would, not, and, had, that
police | guards, guard, military, officials, bainbridge
heavy | punk, pounder, enemy, transformers, destroyer
primarily | have, mainly, shanghai, recently, wurzel
construction | constructed, building, vivid, quarrying, romanesque
applications | systems, operating, computer, os, solaris
existence | eternal, lihou, believes, enlightenment, essence
institute | school, studies, biotechnology, lecturer, arts
square | kilometre, hour, mile, miles, magnificently
...
s | one, four, two, nine, seven
their | would, or, they, have, those
time | first, reputation, several, externalization, during
to | that, would, while, the, this
used | commonly, use, simpler, term, various
he | his, him, was, she, had
so | ourselves, know, that, will, form
over | total, qasr, million, male, ghola
shows | show, dvd, sexy, television, outro
pope | xi, sixtus, iv, vi, vii
pre | revivalist, preferential, achaemenids, sleepers, confucianism
know | you, i, my, we, want
hit | hits, album, puente, restart, hitting
discovered | discovery, discoveries, voyager, pavlova, lithographs
frac | theta, f, iy, sqrt, prod
notes | compositions, coins, octave, concise, metrology
...
most | are, these, than, considered, more
had | was, later, his, were, first
to | only, not, that, they, would
where | delta, sup, bypasses, recoiled, stosunku
that | it, be, not, to, this
he | his, him, doctorate, went, she
however | not, because, required, even, logically
may | jurisdictions, certain, occur, treated, dosage
assembly | parliament, bicameral, council, seats, proportional
egypt | egyptian, pharaoh, syria, mesopotamia, alexandria
frac | theta, f, right, negative, sqrt
woman | marry, fertility, women, wife, god
defense | weapons, cavalry, superiority, israeli, guns
primarily | have, mainly, than, wurzel, displaced
issue | issues, legal, venom, banknotes, bumbling
articles | org, news, reformation, book, documents
...
some | are, many, does, have, such
there | are, is, in, as, years
six | three, four, eight, one, five
a | in, by, but, an, each
been | early, were, have, stipulated, decades
so | ourselves, trypho, nullified, know, dissociative
had | was, later, were, after, his
or | is, are, other, to, with
cost | price, expensive, piracy, usd, coleco
brother | married, sons, son, wife, daughters
running | runs, run, platform, skits, popularity
bible | testament, hebrew, translation, books, biblical
award | awards, awarded, best, achievement, won
quite | stakes, rustic, sheer, fick, blues
lived | ancestral, deists, abductions, sores, claimed
animals | humans, cows, animal, beings, birds
...
six | eight, three, one, five, four
than | less, most, more, much, makes
however | but, that, the, came, chaldees
often | typically, sometimes, usually, effects, horns
nine | one, eight, three, four, seven
when | to, for, could, bulky, get
from | the, in, and, elsewhere, for
their | they, with, themselves, are, as
ocean | atlantic, am, islands, island, coast
proposed | tariff, bang, via, starships, chairmanship
lived | ancestral, sores, abductions, permian, hartsfield
square | mile, occupies, hour, adjacent, thirds
operating | unix, os, extensions, interface, default
troops | army, forces, war, allied, battle
nobel | prize, laureate, physicist, biochemist, chemist
existence | fact, humans, eternal, supernatural, nihilism
...
such | some, and, as, many, are
more | less, and, than, to, are
war | wars, soviet, army, allied, troops
zero | two, three, four, six, eight
system | systems, dortmund, traffic, levels, operating
up | down, them, someone, easily, bolted
new | york, colony, incorporated, dalrymple, symbolized
no | link, there, recombined, total, yes
experience | observable, posture, experiences, ability, panpsychism
hold | ranking, taxpayers, millennialism, muad, jamais
test | cricket, testing, tests, flight, rocket
troops | army, forces, war, allied, armies
marriage | her, marriages, married, divorce, wife
lived | ancestral, sores, abductions, permian, deists
active | association, abuses, organization, include, member
brother | married, sons, son, wife, daughters
...
when | so, to, for, had, start
often | typically, sometimes, usually, variety, rather
such | are, as, also, many, more
all | are, the, every, any, they
for | a, the, is, to, or
states | united, state, u, law, american
their | they, have, those, as, would
at | year, each, winter, where, is
channel | radio, network, tv, bbc, satellite
experience | observable, posture, panpsychism, username, steamboat
active | abuses, mannerheim, include, youth, salut
mainly | primarily, natufian, forestry, sultanates, relatively
square | mile, area, miles, occupies, hour
troops | army, forces, armies, war, attack
stage | actors, concert, kerguelen, oxidizer, walk
ocean | atlantic, coast, am, stations, pacific
...
called | the, which, a, form, are
a | the, is, for, which, or
but | not, because, very, yet, to
so | ourselves, be, do, we, if
war | wars, military, soviet, army, revolution
he | his, him, who, himself, she
or | are, a, with, also, typically
an | a, of, additional, the, eric
accepted | colleague, tischendorf, confirmation, recombined, proposed
taking | rushed, where, indistinguishable, ill, unsure
except | only, asgard, choked, homeomorphic, if
additional | capabilities, an, single, other, discontinued
mainly | primarily, natufian, sultanates, mesothelioma, hollandic
channel | network, tv, radio, bbc, aired
heavy | pounder, during, lighter, bauds, enemy
frac | f, x, theta, sqrt, n
...
will | your, do, to, allows, you
history | links, historical, topics, site, bibliography
state | legislature, university, goodhue, states, community
world | international, history, war, nations, era
nine | one, eight, two, four, seven
for | a, to, and, in, the
people | births, deaths, living, natives, americans
american | born, musician, singer, actor, americans
san | francisco, jose, diego, california, rio
experience | observable, persist, experiences, knowledge, panpsychism
troops | army, forces, armies, soviet, war
marriage | her, marriages, marry, married, divorce
joseph | isbn, geologist, charles, wilhelm, pierre
existence | eternal, essence, concept, belief, beings
channel | tv, satellite, network, radio, aired
governor | minister, chief, cabinet, legislature, appointed
...
over | total, one, zero, three, two
up | down, starts, need, pair, iommi
all | are, any, such, of, only
four | one, five, three, two, six
had | was, after, were, his, later
from | in, of, on, the, and
war | wars, allied, troops, soviet, military
people | americans, natives, births, deaths, citizens
universe | cosmological, cosmology, moonraker, fictional, mcduck
running | run, runs, ran, platform, tokelau
woman | wife, she, marry, women, her
issue | issues, legal, debate, endorsing, referrals
consists | consisting, aleksey, lebrun, harpists, judicial
accepted | colleague, tischendorf, recombined, confirmation, fathers
something | like, really, nothing, dispel, algorithmically
file | files, disk, format, document, operating
...
d | b, seven, one, nine, eight
three | four, one, two, six, five
an | and, to, a, of, second
at | university, where, failed, an, tour
zero | two, four, three, six, five
use | used, can, systems, using, uses
its | the, tourism, which, commercial, steadily
two | zero, three, four, one, nine
applications | systems, application, technology, software, interface
taking | where, unsure, indistinguishable, take, attending
except | only, there, not, all, nouns
mathematics | mathematical, theory, algebra, theorems, numbers
lived | he, died, sores, mother, man
professional | players, sports, cornerbacks, association, trained
liberal | conservatism, party, conservative, progressive, politics
troops | army, war, battle, allied, soviet
...
on | of, the, with, and, between
three | two, four, one, six, nine
up | down, promenade, starts, almost, you
united | states, british, nations, britain, australia
no | link, import, if, duplicate, already
also | see, list, as, main, known
was | had, later, his, after, died
new | york, press, reprinted, acre, isbn
question | questions, whether, call, if, skeptical
mean | temperature, diameter, average, parallax, unobservable
dr | dre, sara, report, trevor, theodor
creation | created, explanation, flood, worlds, anatomists
freedom | rights, restoring, expression, sustainability, political
articles | com, article, books, book, org
animals | humans, animal, eating, reproduction, cows
behind | thrust, quickly, belichick, preventing, side
...
can | be, example, are, must, means
over | total, four, zero, three, nine
may | be, any, certain, this, or
years | year, months, age, five, one
these | are, many, most, not, such
if | we, be, every, any, then
first | by, later, early, which, in
on | the, of, from, with, and
universe | cosmological, cosmology, moonraker, matter, fictional
animals | humans, animal, eating, mammals, cows
governor | representative, legislature, appointed, government, chief
active | secretariat, mannerheim, presides, members, abuses
scale | scales, observations, instrument, quantities, large
arts | art, graduate, institutions, degree, prestigious
shown | ni, alene, delineate, exponential, stress
joseph | composer, jeffery, painter, honor, scarpa
...
d | b, e, f, c, pr
are | or, such, other, these, all
that | to, be, any, a, this
where | delta, same, exists, if, is
system | systems, the, operating, separate, which
will | allows, does, my, can, wait
zero | two, three, four, one, five
use | used, can, using, many, uses
resources | resource, information, agriculture, source, external
ice | hockey, canadian, athletes, roller, basketball
recorded | records, dates, dated, larmor, coptic
placed | resting, tosses, redeemed, somewhere, merciful
derived | modern, derives, phrase, root, distinct
mainly | primarily, parts, some, mountainous, language
ocean | atlantic, islands, pacific, intelsat, am
award | awards, awarded, best, won, emmy
...
would | to, they, their, had, that
four | one, two, five, three, seven
most | have, many, particularly, include, such
many | such, some, most, these, were
people | deaths, natives, living, americans, citizens
eight | one, six, seven, four, five
into | the, part, east, in, to
his | he, him, himself, after, was
pressure | liquid, temperature, pressures, propellant, heat
assembly | council, parliament, deputies, legislative, member
units | unit, prefixes, disengage, approximately, force
instance | cobol, languages, jurisdictions, derivation, addition
defense | security, foreign, agency, u, service
know | you, want, i, me, we
mainly | primarily, mountainous, some, parts, mostly
ocean | atlantic, islands, pacific, stations, coast
...
seven | one, four, five, eight, three
would | to, their, they, that, had
eight | six, four, one, five, three
two | zero, three, four, one, five
his | he, him, had, himself, after
states | united, u, state, constitution, virginia
three | two, four, one, six, five
new | york, press, nine, state, conference
powers | exercised, power, wigner, expanded, fraught
account | accounts, definitive, mentions, discordia, executables
instance | jurisdictions, use, languages, borrowings, example
bbc | interview, abc, aired, radio, news
numerous | including, and, several, include, are
operating | unix, os, ported, windows, extensions
cost | dollars, budget, expensive, costing, invest
report | reports, concluded, pdf, witness, coop
...
if | example, we, not, true, measure
use | used, systems, using, are, uses
all | every, with, be, are, such
are | or, such, other, these, there
often | typically, sometimes, usually, settings, rather
seven | one, four, five, eight, six
over | zero, total, four, five, two
may | be, certain, depending, occur, specific
proposed | rejected, tariff, formed, bill, proposal
powers | exercised, monarch, power, exercising, ceremonial
existence | eternal, universe, reincarnation, discovery, lihou
operating | unix, systems, ported, microsoft, windows
hit | hits, album, hitting, billboard, debut
account | definitive, accounts, executables, discordia, meteorologists
applications | software, interface, systems, application, proprietary
shows | show, appeared, television, itv, theatrical
...
i | you, my, me, t, know
b | d, actor, one, writer, politician
may | certain, depending, occur, or, should
two | three, zero, four, nine, five
the | of, is, a, and, by
world | between, history, in, most, total
for | a, and, is, with, only
and | of, with, for, in, the
resources | resource, hydropower, protection, agricultural, petroleum
quite | difficult, different, most, straightforward, than
centre | centers, museums, airport, shopping, education
ocean | pacific, atlantic, am, intelsat, islands
animals | animal, humans, eating, species, reproduction
derived | derives, root, origin, corresponding, latin
units | unit, si, prefixes, conversion, approximately
versions | version, proprietary, systems, interface, portable
...
years | year, months, age, days, over
where | delta, then, mathbf, right, left
so | do, that, can, it, able
such | are, or, as, called, used
known | of, also, the, other, referred
one | four, three, eight, seven, nine
but | just, however, to, did, not
b | d, actor, politician, writer, one
quite | difficult, mechanically, different, used, straightforward
alternative | many, such, hdl, standard, annotate
channel | channels, radio, network, satellite, tv
know | you, say, me, myself, stuff
road | roads, route, airport, connecting, passenger
mainly | stretched, almost, mostly, mountainous, mixed
grand | michigan, street, virginia, soccer, prix
test | testing, tests, flight, cricket, risk
...
zero | two, five, eight, three, six
about | approximately, thousand, half, million, estimated
however | this, that, been, fact, although
used | use, usually, other, uses, term
their | themselves, they, them, only, would
had | were, later, he, went, after
its | steadily, due, far, tourism, southwest
have | are, been, that, some, they
bill | bush, singer, george, clinton, president
engineering | technology, engineers, technical, electrical, physics
older | age, median, families, household, younger
accepted | was, universally, be, tischendorf, papal
assembly | elected, deputies, parliament, senate, seats
proposed | rejected, proposal, completed, tariff, aaaa
mean | value, diameter, variable, arithmetic, approximated
account | discoveries, surpluses, infliction, accounts, explanation
...
war | battle, allied, troops, army, defeat
eight | one, six, seven, five, nine
four | one, two, five, six, three
can | properties, is, example, requires, used
more | less, than, tend, or, usually
was | later, had, a, were, died
as | be, the, also, known, not
in | the, of, and, first, all
ice | canadian, snow, glacier, covered, snout
prince | princess, daughter, married, emperor, assassinated
applications | application, software, technologies, postscript, desktop
engine | engines, turbine, powered, fuel, cars
bbc | april, links, listing, news, march
http | www, html, com, htm, org
orthodox | church, catholic, orthodoxy, churches, eastern
bill | bush, clinton, president, singer, senator
...
american | actor, musician, actress, singer, born
years | year, age, months, period, six
they | are, them, themselves, those, their
from | in, the, elsewhere, and, of
zero | two, one, five, six, four
b | d, politician, actor, musician, physicist
to | higher, the, but, this, or
than | less, most, more, higher, relatively
question | questions, whether, answer, asked, how
active | organization, enhance, association, nations, include
defense | security, force, air, armed, mounted
numerous | including, most, are, were, various
applications | software, application, systems, desktop, interface
units | unit, infantry, prefixes, kilometre, comparison
notes | note, coins, concise, subdominant, edition
pre | savages, ceremonial, celta, ordovician, been
...
they | themselves, all, their, to, are
would | that, but, eventually, because, had
has | is, the, that, arguably, some
people | americans, living, births, deaths, natives
can | cannot, or, useful, do, be
so | to, i, will, able, do
five | one, six, four, three, eight
and | in, of, the, by, one
cost | dollars, expensive, usd, costs, price
running | run, next, runs, pick, workstations
writers | novelists, poets, philosophers, births, fiction
road | roads, peachtree, route, street, track
something | you, wrong, like, replied, might
lived | story, exile, mother, least, wife
egypt | egyptian, pharaoh, syria, mesopotamia, ancient
magazine | interview, fiction, magazines, weekly, news
...
one | eight, four, nine, five, seven
his | him, himself, he, father, wrote
he | his, him, himself, she, father
their | they, themselves, have, more, these
three | two, five, four, one, six
to | by, and, the, not, or
s | nine, five, eight, four, two
used | use, usually, uses, common, widely
institute | university, research, institutions, college, school
nobel | prize, laureate, physicist, physiology, chemist
except | only, so, gon, if, are
engineering | engineers, sciences, engineer, technology, electrical
square | adjacent, mile, sq, occupies, city
cost | dollars, expensive, usd, costs, budget
award | awards, awarded, best, nominated, achievement
san | francisco, diego, jose, santa, el
...
called | which, a, the, name, also
at | the, times, three, lengths, winter
other | are, or, all, similar, types
he | his, him, she, himself, who
are | or, other, these, all, types
many | different, some, these, are, such
years | age, zero, six, four, five
th | century, nd, centuries, rd, late
liberal | conservatism, conservative, party, political, democratic
notes | note, coins, fifth, edition, concise
pressure | temperature, pressures, liquid, fluid, phase
recorded | records, song, albums, album, music
ice | snow, cream, melted, warm, glaciations
heavy | armoured, lighter, armor, enemy, armour
something | you, wrong, knows, look, get
bill | jim, david, george, bills, evans
...
would | eventually, that, did, it, had
used | use, using, commonly, as, usually
at | the, times, begins, newark, year
where | mathbf, then, delta, dense, left
two | three, five, four, zero, one
its | a, the, commercial, far, steadily
who | he, him, whom, young, father
time | when, minimization, place, externalization, redstone
quite | different, very, humans, difficult, straightforward
marriage | marriages, marry, divorce, married, polygamy
hit | hits, hitting, song, billboard, album
nobel | prize, laureate, physiology, physicist, chemist
universe | universes, cosmological, cosmology, cosmic, bang
freedom | sustainability, freedoms, ethics, rights, chorionic
pre | celta, americas, savages, splice, curricular
road | roads, route, peachtree, street, highway
...
be | that, should, can, not, cannot
war | wars, wwii, allied, soviet, military
but | would, not, that, did, they
two | three, zero, four, five, one
was | october, had, after, early, february
new | york, press, nine, seven, state
history | historical, links, site, references, museum
d | b, writer, eight, politician, composer
account | beg, explanation, infliction, imbalance, chapter
mathematics | mathematical, theoretical, algebraic, calculus, axiomatic
dr | presenter, theodor, scientist, ketcham, rick
discovered | discovery, discoveries, discoverer, later, ago
troops | forces, armies, soviet, confederacy, command
stage | actors, theatre, broadway, film, musical
arts | art, graduate, students, degree, theatre
question | questions, whether, answer, that, answers
...
its | a, the, due, steadily, and
two | three, one, zero, four, nine
where | delta, sum, right, mathrm, if
years | year, age, over, six, female
nine | one, two, eight, four, six
about | million, thousand, and, around, estimated
their | they, have, but, them, that
called | the, name, glutamic, form, a
bill | clinton, jim, musician, evans, jennifer
prince | princess, crown, throne, king, abdication
smith | thomas, jim, coordinator, braxton, john
primarily | mainly, and, major, groups, related
channel | tv, channels, aired, satellite, radio
mean | signifies, means, meaning, does, variance
report | pdf, reports, international, news, biggio
discovered | discovery, discoveries, discoverer, ago, named
...
they | are, their, those, them, do
these | are, be, other, many, have
nine | one, seven, eight, two, five
war | defeat, troops, wars, battle, allied
between | the, which, through, of, shared
may | or, should, does, indefinite, take
three | two, six, five, one, four
is | of, the, a, or, if
applied | ethics, which, electrical, dynamics, disciplines
proposed | rejected, proposal, formed, independently, formulated
magazine | magazines, weekly, newspaper, interview, october
ice | canadian, hockey, basketball, snow, roller
woman | married, she, women, fertility, herself
existence | eternal, exist, universes, beings, notion
lived | died, he, mother, his, was
operations | operation, assignment, security, boolean, arithmetic
...
are | other, there, these, have, all
up | down, easiest, quickly, ballcarrier, so
d | b, j, composer, writer, politician
see | list, links, also, external, article
may | occur, or, should, does, do
also | as, see, of, list, including
this | not, which, it, have, could
nine | one, eight, seven, four, five
mainly | primarily, also, industry, persians, mountainous
notes | note, coins, bass, compositions, harmony
derived | word, modern, derives, etymology, distinct
defense | defence, branches, civil, defensive, offensive
freedom | rights, freedoms, ethics, liberty, liberties
applications | systems, application, interface, software, technologies
recorded | records, yielded, released, tune, observations
hit | hits, hitting, album, swing, record
...
time | year, months, periods, the, years
about | approximately, estimated, thousand, than, estimates
or | other, is, be, are, often
has | have, which, is, some, been
so | how, it, even, you, have
at | of, in, the, after, three
but | him, of, later, was, the
one | eight, nine, three, four, seven
notes | note, coins, editions, compositions, metrology
proposed | proposal, rejected, accepted, formed, independently
nobel | prize, laureate, physicist, physiology, chemist
award | awards, awarded, best, oscar, won
older | families, household, median, younger, females
police | military, enforcement, officials, department, battalion
something | you, i, knows, lot, seem
derived | derives, modern, word, is, root
...
th | centuries, century, nd, late, around
after | before, had, until, was, during
where | go, left, y, or, delta
world | nations, international, country, asia, organization
new | york, nine, corporation, june, albany
there | are, have, common, also, only
s | one, nine, two, five, six
called | is, also, containing, the, are
engine | engines, fuel, turbine, powered, combustion
active | bind, isoamyl, support, tightly, secretariat
dr | theodor, john, actor, ketcham, seymour
grand | knights, michigan, highwind, commemorating, prix
professional | teams, tournaments, sports, trained, players
heavy | alloys, metal, armor, lighter, industries
road | roads, passenger, railway, interstate, city
articles | article, sites, external, list, org
...
five | one, eight, four, three, two
by | of, the, one, and, a
world | international, nations, history, era, during
i | my, you, me, am, we
many | most, such, these, are, notable
into | through, the, directly, them, in
years | year, remaining, months, age, days
two | zero, three, four, one, five
hit | hits, hitting, album, record, singles
notes | note, bass, coins, compositions, editions
test | testing, tests, flight, cricket, concorde
resources | resource, protection, help, deposits, arable
account | accounts, billion, book, ambrosius, beg
construction | constructed, bridge, bridges, tunnel, building
shows | show, series, television, appearance, icp
existence | that, exist, eternal, supernatural, concept
...
also | as, are, and, with, most
four | three, one, two, five, six
by | the, and, of, other, which
such | are, many, most, usually, or
world | international, country, nine, nations, during
a | the, as, is, in, and
which | the, and, in, of, is
first | was, second, one, seven, the
rise | centuries, arose, conquest, eastern, culminating
smith | jewell, coordinator, braxton, greg, brett
award | awards, awarded, best, oscar, winners
paris | france, le, les, fran, french
experience | experiences, thinking, perception, awareness, soul
lived | return, briefly, he, mother, have
alternative | many, oldies, major, like, complementary
writers | novelists, winners, fiction, authors, literature
...
eight | one, five, six, four, seven
however | although, few, not, that, but
is | a, are, if, or, example
all | are, they, every, define, with
five | four, one, six, two, three
so | you, really, if, pull, that
from | in, the, on, across, to
been | discovered, samsa, drying, many, have
dr | scientist, actor, protagonist, theodor, d
freedom | freedoms, liberty, rights, individual, liberties
lived | briefly, return, he, went, wealthy
ice | hockey, snow, roller, canadian, glaciation
centre | centers, main, centres, airport, universities
nobel | prize, laureate, physicist, physiology, chemist
bbc | abc, itv, uk, interview, news
governor | legislature, chief, minister, cabinet, appointed
...
d | b, composer, writer, politician, one
many | most, have, various, some, these
i | you, my, t, me, am
as | of, also, a, and, the
five | four, one, six, two, three
there | are, is, and, all, of
who | he, had, slaying, whom, while
about | estimated, article, country, accounts, has
professional | trained, teams, team, athletic, sports
additional | pdf, extra, number, added, rare
derived | derives, etymology, latin, word, use
bill | clinton, bush, politician, governor, senator
writers | novelists, winners, fiction, literature, deaths
discovered | discovery, discoveries, named, mysterious, found
brother | son, throne, married, daughters, father
operations | operation, squadron, combat, evaluating, reconnaissance
...
in | the, and, of, first, a
other | are, or, such, most, include
about | how, article, related, answer, has
called | a, the, are, of, include
some | have, are, these, or, that
eight | six, four, one, five, three
four | one, three, two, eight, nine
and | of, in, the, for, are
older | household, families, median, younger, age
police | arrested, officers, enforcement, guards, military
hit | hits, hitting, got, album, career
freedom | freedoms, rights, liberty, recipients, liberties
award | awards, awarded, nominations, best, won
quite | difficult, used, enough, manufacturers, especially
defense | defence, tactics, military, tribunal, air
san | francisco, diego, california, antonio, da
...
who | he, whom, while, people, slaying
will | shall, cannot, does, you, your
no | true, there, cannot, condoned, said
for | and, a, all, two, of
such | are, these, other, modern, with
used | use, usually, uses, commonly, using
up | down, take, walked, get, you
time | already, after, unprecedented, failed, hours
applications | software, application, server, desktop, interface
accepted | regard, recognized, marxist, not, remained
bible | testament, tanakh, biblical, scripture, prophets
arts | art, styles, school, technique, graduate
assembly | council, elected, legislative, deputies, senate
rise | arose, almohades, nazism, centuries, occupation
pressure | pressures, fluid, gases, liquid, propellant
lived | missionary, ranch, wealthy, settled, returned
...
its | far, the, was, due, which
new | york, company, sold, philadelphia, state
they | them, to, are, not, those
had | later, was, he, after, were
would | not, that, eventually, even, did
it | a, not, that, been, to
be | not, that, can, only, no
this | the, however, truism, interruptus, believe
cost | expensive, costs, dollars, price, usd
versions | version, models, compatibility, compiler, graphical
additional | extra, mode, provide, added, special
shows | show, animated, television, episodes, sitcom
freedom | freedoms, rights, moral, ethics, recipients
applications | software, application, desktop, server, os
dr | scientist, actor, musician, rapper, pseudonym
creation | monotheism, creator, abrahamic, judaism, destiny
...
if | we, unless, condition, so, let
often | typically, variety, usually, use, these
between | geographic, dispute, boundary, the, both
with | and, are, in, as, a
or | either, are, typically, can, other
new | york, philadelphia, opens, april, sold
six | eight, four, five, one, three
more | less, than, tend, very, although
except | but, dialects, nearly, joins, enumerated
existence | supernatural, exist, argument, belief, humans
creation | monotheism, creator, judaism, abrahamic, idea
liberal | conservative, democrats, liberals, democratic, conservatives
shown | showed, mg, aleph, thyroid, tnf
prince | princess, throne, iii, son, crown
bill | bills, clinton, bush, linda, impeachment
universe | fictional, universes, cosmology, character, cosmological
...
than | less, most, tend, size, higher
or | either, typically, are, usually, can
between | straightened, of, side, correspondence, boundary
first | second, the, one, three, in
can | or, types, cannot, be, may
also | as, especially, a, list, are
from | and, in, the, down, of
up | down, with, while, back, went
ice | hockey, canadian, glaciers, snow, glaciations
lived | story, exile, briefly, missionary, return
hit | hits, hitting, recorded, swing, charts
orthodox | catholic, churches, orthodoxy, church, christianity
mean | sqrt, variance, approximated, temperature, frac
accepted | credibility, tischendorf, universally, hypothesis, date
derived | word, meaning, derives, originated, modern
joseph | smith, latter, adolf, jesus, austrian
...
that | some, not, be, because, might
used | use, for, uses, other, usage
first | second, the, early, was, one
it | is, does, which, not, even
while | by, along, were, their, kwan
time | hour, lasted, already, during, after
who | whom, himself, he, father, illegally
six | eight, five, one, four, nine
pre | americas, conquered, columbian, early, oriental
behind | federals, front, back, yard, defensive
woman | fertility, male, she, pregnant, female
square | adjacent, sq, squares, mile, city
resources | resource, environmental, arable, of, agricultural
older | median, families, age, household, versus
rise | elevations, peak, expanded, plains, rising
award | awards, awarded, best, achievement, prize
...
i | you, yes, t, sorry, am
from | the, in, of, down, at
only | to, except, not, be, but
over | years, zero, total, three, period
b | d, r, v, c, y
four | three, one, two, five, six
on | the, of, be, kayaking, from
during | before, early, became, period, was
applications | software, application, solutions, useful, computer
know | you, my, want, think, how
powers | governor, territory, commander, responsibilities, judicial
joseph | smith, latter, jesus, composer, william
orthodox | churches, orthodoxy, christians, catholicism, catholic
prince | throne, princess, son, abdication, abdicated
grand | ceded, lithuania, croix, prix, michigan
heavy | armor, lighter, armoured, heavier, vehicles
...
on | the, a, also, of, introducing
it | not, even, a, that, is
nine | one, eight, two, seven, three
state | states, colleges, new, united, university
seven | one, three, eight, five, nine
is | a, in, are, be, also
their | they, them, these, but, to
with | or, in, the, such, a
powers | empire, power, commander, territory, governor
applications | software, application, technologies, solutions, computers
shown | showed, tnf, shows, aleph, have
derived | word, etymology, latin, root, meaning
animals | animal, eating, humans, mammals, organisms
engineering | technology, engineers, engineer, management, technical
additional | extra, are, provide, option, added
prince | throne, princess, son, abdication, abdicated
...
from | in, down, on, and, the
their | they, them, these, have, would
at | three, where, time, two, one
eight | one, seven, nine, six, five
where | at, then, formula, the, this
three | four, one, five, two, six
five | one, three, four, six, eight
after | had, during, last, before, months
engineering | technology, engineers, management, engineer, research
grand | croix, prince, prix, ceded, of
woman | daughter, she, her, female, women
articles | external, article, com, links, site
applications | application, software, systems, server, technologies
report | pdf, reports, news, commission, review
existence | entity, notion, property, exist, conception
numerous | were, bizarre, have, most, including
...
are | have, or, other, some, such
war | wars, defeat, army, civil, independence
so | that, not, do, be, have
or | are, may, which, either, of
on | from, the, of, in, with
that | not, be, this, would, cannot
they | them, do, not, have, some
five | one, eight, four, three, six
woman | she, her, daughter, women, married
ocean | pacific, atlantic, islands, coast, oceans
bill | clinton, bills, jim, joe, bush
writers | novelists, poets, fiction, deaths, academics
instance | specific, which, value, yoghurt, androgens
derived | word, derives, etymology, modern, origin
institute | university, technology, research, college, school
know | you, want, didn, my, i
...
can | be, example, need, cannot, above
will | shall, does, they, allows, wait
had | was, he, his, after, she
years | year, age, months, over, days
an | is, which, the, of, a
which | a, the, is, its, has
many | some, other, most, different, are
there | are, have, that, some, or
magazine | newspaper, magazines, weekly, interview, comics
governor | minister, appointed, senate, senators, president
primarily | northerly, mainly, major, includes, most
discovered | discovery, discoveries, colonized, discoverer, asteroid
running | run, runs, ran, emulation, line
cost | expensive, costs, price, dollars, usd
prince | princess, crown, throne, abdicated, emperor
bill | clinton, impeachment, bills, jim, alex
...
there | are, have, all, that, only
not | be, do, that, it, must
on | the, of, with, from, between
used | use, uses, be, are, term
that | not, be, existence, thus, a
at | in, where, the, on, observatory
as | the, also, well, including, of
people | natives, americans, citizens, minority, ethnic
articles | discussion, online, publications, books, article
arts | art, martial, graduate, styles, sports
alternative | annotate, authored, hypothesis, trends, scientifically
bbc | news, itv, allafrica, tv, aired
joseph | composer, austrian, botanist, johann, von
freedom | freedoms, rights, liberty, liberties, liberalism
road | roads, highway, railroad, peachtree, track
mathematics | mathematical, theory, axiomatic, physics, algebra
...
where | at, chili, is, left, or
the | of, in, and, a, by
has | have, is, which, such, not
in | the, of, and, which, one
be | not, that, cannot, can, to
other | are, some, etc, or, kinds
it | not, that, could, this, do
can | are, be, cannot, must, if
frac | cdot, sqrt, infty, cos, mathbf
arts | art, students, graduate, yale, school
shows | show, television, nbc, tv, comedy
test | testing, tests, simulate, cricket, detection
cost | costs, expensive, usd, dollars, wages
san | francisco, santa, diego, jose, los
smith | john, greg, jim, graeme, ian
grand | prize, marshals, croix, michigan, knights
...
often | sometimes, or, usually, are, form
some | are, many, or, those, and
b | d, c, p, f, y
his | he, him, himself, father, life
used | use, example, uses, usage, or
see | links, list, external, uses, also
nine | four, seven, one, two, five
be | are, not, or, that, can
know | you, how, want, sure, i
stage | musicals, stages, actors, broadway, synchronized
powers | power, exercised, constitutional, government, retains
behind | face, front, side, downward, defensive
pressure | pressures, cardiac, pump, subjected, explosive
running | emulation, ran, runs, unix, quarterbacks
shown | are, tnf, aleph, different, pronunciation
smith | john, greg, jim, graeme, jewell
...
this | kalmyk, is, the, be, when
eight | one, six, seven, four, five
may | must, or, certain, can, be
states | united, us, state, u, admitted
during | year, after, days, months, lasted
nine | one, four, six, two, five
are | other, or, usually, such, is
to | would, too, when, they, only
accepted | universally, recognized, stated, accepts, not
consists | consisting, comprises, separated, composed, consist
alternative | many, annotate, well, non, embraced
resources | resource, contribute, reserves, deposits, protection
derived | derives, modern, etymology, root, similar
pope | papacy, papal, vii, king, xii
versions | version, windows, operating, graphical, vendors
hold | holding, entire, holds, whether, deny
...
into | once, the, in, first, which
is | the, of, a, are, or
from | in, and, the, on, which
are | other, usually, such, or, is
that | not, be, it, to, did
th | century, nd, rd, six, in
there | these, still, are, have, common
was | after, became, a, later, he
universe | cosmology, cosmological, galaxy, fictional, bang
award | awards, awarded, best, emmy, pulitzer
pope | papal, papacy, xii, pius, benedict
something | you, audience, really, like, wrong
except | dialects, be, inhabited, pronounce, if
smith | mormon, jewell, primus, fatou, greg
recorded | records, recording, song, album, recordings
issue | issues, nullification, reconciling, racism, commenting
...
known | also, called, century, as, referred
between | differences, arises, divided, with, relationship
nine | one, six, four, three, two
american | actor, writer, british, americans, comedian
that | be, it, is, cannot, a
six | eight, five, four, one, nine
during | occurred, after, lasted, nine, days
up | down, in, out, the, from
cost | costs, inexpensive, expensive, pricing, dollars
pre | hagia, americas, periods, columbian, thr
award | awards, awarded, emmy, pulitzer, winner
question | questions, asked, answer, answers, what
professional | teams, sports, trained, amateur, team
egypt | egyptian, cairo, egyptians, suez, arab
event | events, sporting, week, extinction, sukkot
articles | publications, online, news, books, org
...
while | to, had, their, only, who
five | four, three, six, one, eight
some | many, are, these, such, have
its | the, part, has, which, area
i | you, my, t, am, yes
by | the, of, one, and, as
after | was, had, during, in, before
with | and, the, in, for, of
experience | experiences, perception, sensory, awareness, psychotherapy
institute | technology, university, college, research, school
writers | novelists, winners, authors, fiction, poets
brother | son, throne, daughter, married, younger
paris | fran, france, les, du, lyon
engineering | technology, engineers, graduate, institute, professor
http | www, html, htm, com, edu
bill | clinton, bills, submitted, legislation, politician
...
not | they, do, to, does, need
most | many, these, include, considered, other
seven | one, three, four, five, two
american | actor, writer, singer, composer, musician
its | it, the, which, by, part
four | two, three, one, five, eight
state | governor, united, government, federal, governors
united | states, governors, state, north, presidents
engineering | technology, engineers, institute, physics, technical
operating | unix, windows, software, desktop, bsd
bbc | february, listing, day, march, leap
paris | france, fran, de, le, du
account | accounts, imbalance, predictions, value, pliny
animals | animal, mammals, insects, cows, phylum
accepted | universally, regard, rejected, accept, recognized
construction | constructed, freight, tunnel, transportation, railroad
...

Visualizing the word vectors

Below we'll use T-SNE to visualize how our high-dimensional word vectors cluster together. T-SNE is used to project these vectors into two dimensions while preserving local stucture. Check out this post from Christopher Olah to learn more about T-SNE and other ways to visualize high-dimensional data.



In [14]:

    
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE



In [16]:

    
# getting embeddings from the embedding layer of our model, by name
embeddings = model.embed.weight.to('cpu').data.numpy()



In [26]:

    
viz_words = 600
tsne = TSNE()
embed_tsne = tsne.fit_transform(embeddings[:viz_words, :])



In [27]:

    
fig, ax = plt.subplots(figsize=(16, 16))
for idx in range(viz_words):
    plt.scatter(*embed_tsne[idx, :], color='steelblue')
    plt.annotate(int_to_vocab[idx], (embed_tsne[idx, 0], embed_tsne[idx, 1]), alpha=0.7)



In [ ]: